Key Points 


Virtual schools and alternative K-12 education models are typically evaluated using 
traditional accountability metrics despite having distinct student populations and edu- 


cational missions. 


A new accountability model is needed for these schools that provides regulators with 
the data they need to ensure a baseline level of education and families with information 


that will help them compare alternative education options. 


Metrics for alternative school accountability should be outcomes-based, adaptive to 
schools’ particular missions, and audited to guard against manipulation. 


As full-time virtual learning moved from the fringe 
to the mainstream in mid-March 2020, onlookers 
expressed concerns about access for learners and 
accountability for the learning itself. 

Since at least 2015—-when the widely cited 
National Study of Online Charter Schools by the 
Center for Research on Educational Outcomes 
(CREDO) at Stanford University, Mathematica 
Policy Research, and the Center for Reinventing 
Public Education found that students enrolled in 
full-time, virtual charter schools made, on average, 
far less progress than their counterparts in tradi- 
tional schools did'—educators of all stripes have 
had significant concerns around the quality of 
these offerings. 

These concerns haven’t applied only to full- 
time virtual schools, however. Alternative schools 
that serve students who have dropped out or trans- 
ferred from traditional schools—and often serve as 
schools of last resort—have historically struggled 
to show the value they add to students. They have 
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also struggled to allow families and regulators to 
make sound judgments around their value. Alt- 
hough supporters suggest that these schools are 
crucial because they provide smaller classes, more 
flexible schedules, and more counseling, the few 
studies that have looked specifically at alternative 
schools have documented significant quality con- 
cerns, low graduation rates, and underfunding.” 

The question for regulators, families, and the 
schools themselves is whether these alternative 
schools have poor outcomes because of the stu- 
dents they serve and their resources or because 
they are poorly serving these students. The reality 
on the ground is surely mixed. But the data 
states provide—coupled with the diversity of the 
alternative school sector and the high proportion 
of at-risk students it serves—make it difficult to 
disentangle.3 

This report argues that alternative schools need 
a new set of standard metrics to help families and 
policymakers gauge their quality. These metrics 


should be based on student outcomes—not inputs. 
And they should allow schools some latitude to 
select the measurements most applicable to their 
mission. However, those measurements must also 
have enough in common that they give prospective 
families a clear view of whether a school is the right 
fit for them and policymakers greater clarity as to 
whether the school is serving its students well and 
fulfilling its mission. 


Current Efforts Around Alternative and 
Virtual Education Accountability Are 
Inadequate 


A few states have tried to address concerns 
about the quality of alternative schools. A review 
of 38 states’ Every Student Succeeds Act plans by 
the American Youth Policy Forum, for instance, 
found that eight states’ plans distinguish between 
traditional and alternative schools by adding inter- 
vention or support strategies, using GED rates or 
growth rates on assessments instead of profi- 
ciency on traditional end-of-year tests, or modify- 
ing accountability measures for alternative schools. 
Mostly, however, alternative schools are evaluated 
using the same metrics as traditional schools despite 
serving a different population.4 

Alternative schools themselves have attempted 
to rewrite the narrative around alternative educa- 
tion by developing their own accountability frame- 
works. In Massachusetts, for example, the few 
alternative charter schools that exist and serve 
nontraditional students have created an alterna- 
tive set of measures to monitor their quality, as 
they contend that the traditional metrics (e.g., sum- 
mative test scores and on-time graduation rates) 
don’t make sense to measure the outcomes for stu- 
dents who enter the schools overage and under- 
credited.s One of these schools, the Boston Day 
and Evening Academy, has a set of measures related 
to the execution of its mission—as opposed to its 
outcomes—that monitors the extent to which its 
school culture is “safe and equitable,” learning is 
competency-based, assessments are “student- 
centered,” and postgraduate planning is available.° 

Full-time virtual charter schools have similarly 
denounced the methodology behind research at 
centers like CREDO by suggesting that it paints a 
flawed picture of their schools’ performance.’ Like 
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the Boston Day and Evening Academy, they cor- 
rectly claim that traditional accountability metrics 
don’t capture the circumstances in which their stu- 
dents enroll and therefore do not capture the value 
they provide. 

To illustrate one such situation that virtual 
schools might confront: Imagine a school serving 
students entering at a fifth-grade age but with sec- 
ond-grade math abilities. The school chooses to 
serve the students where their learning needs are, 
not where the traditional system thinks they 
should be. Suppose that by the end of the year the 
students are performing math at a fourth-grade 
level. This would seem a remarkable success, but 
the school would likely not get credit for it under 
most accountability regimes today, as the tests 
focus narrowly on grade-level material as opposed 
to understanding the starting and ending points of 
a given student. This situation of course also applies 
to traditional schools, but a striking number of the 
students enrolled in alternative arrangements falls 
under patterns like this one. 


Why There Must Be a New Framework 
to Measure Quality 


Throwing darts at existing metrics and studies 
isn’t enough. Nor are one-off accountability 
frameworks like the one Boston Day and Evening 
created, which fail to provide not only families 
with the information they need to make judgments 
between alternative schools but also regulators 
with the data they need to ensure a baseline level 
of education. Even current state-based frame- 
works that acknowledge the difference between 
alternative and traditional schools lack a common 
philosophy for representing student outcomes. 
Instead, a uniform framework that allows fami- 
lies and regulators to make judgments around 
schools’ relative quality is needed so they can make 
decisions that will ultimately hold these schools 
accountable. Such a new model to measure quality 
should not focus on a school’s activities—such as 
the time spent learning, who works with students, 
the use of “best practices,” and the recruitment of 
students that fit their charter. (This is what alter- 
native schools that have worked with their state to 
produce alternative accountability frameworks 
have historically done.)® New transparency models 


should instead focus on outcomes relative to the 
students each school serves—and thus focus on 
objectives tailored to the school’s mission—with 
clear data that are audited by an independent third 
party. 

Focusing on inputs locks institutions into a 
fixed way of doing things and inhibits their ability 
to innovate. Focusing on outcomes, however, 
encourages continuous improvement toward an 
overall set of goals—in this case, goals around stu- 
dent success. And it gives parents and policymak- 
ers a baseline to evaluate the schools. 

Not dictating input-based measures for institu- 
tions to follow—such as seat-time measures, class- 
size numbers and configurations, and the creden- 
tials that teachers must hold—allows for novel and 
unconventional school models that parents can 
choose in accordance with their preferences and 
needs, and it will allow policymakers to see 
whether students are excelling in line with the 
public interest. Although reporting on inputs can 
help parents find a school that matches a certain 
educational philosophy and therefore has value as 
a tool for families, it falls short of helping all stake- 
holders understand the outcomes from those mod- 
els so they can make decisions around enrolling 
and regulatory approval. 


A Path Forward 


The Education Quality Outcomes Standards 
Board, a nonprofit I helped found, has put forth a 
framework that, although focused on postsecond- 
ary education, can begin to offer a set of standards 
for measuring different kinds of outcomes based 
on institution type at the K-12 level. The frame- 
work calls for a two-pronged approach. First, it 
establishes standards in five different elements’®: 
learning outcomes, program completion, place- 
ment upon leaving, postgraduation earnings, and 
student satisfaction. Not all these elements will 
apply to all schools, as some are a better fit for 
schools that have a career and technical focus and 
serve older students. In other words, a school’s 
mission and who it serves would shift the indica- 
tors in the standards used. Although no system can 
capture perfectly every dimension of an education 
program’s quality, these standards allow institu- 
tions to report outcomes through the lens of the 
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students. To that end, the standards allow schools 
to report on a range of near- and medium-term 
outcomes related to learning, completion, place- 
ment, earnings, and satisfaction that each program 
could claim to provide. 

Second, an auditor would conduct an examina- 
tion to provide assurance that the outcomes reported 
were collected and reported in accordance with the 
standards spelled out in the Education Quality 
Outcomes Standards Board framework and that 
there is no evidence of manipulation. The inten- 
tion, ultimately, is that the schools report data on 
the claims they make about the value they contrib- 
ute to students and society. 


New transparency models should 
instead focus on outcomes relative 
to the students each school serves. 


If a program claims to provide a career benefit— 
as many career and technical schools do—that 
school would measure its benefit using such indi- 
cators as placement in jobs, earnings, and student 
passing rates on relevant certifications, which 
would allow stakeholders to compare the program 
to alternative options. Although different schools 
might report different sets of outcomes in line with 
the value they provide students and their pro- 
grams’ missions, a functional quality assurance 
system should rely on multiple measures of results. 


Five Quality Assurance Elements for Alterna- 
tive Schools. The next section summarizes how a 
K-12 alternative school could think about report- 
ing outcomes in each of the five categories. All 
these measures should be broken down by various 
subgroups that are demographic and capture 
student and parent intention in enrolling and the 
academic level on entry, as discussed below. 


Learning Outcomes. In addition to having students 
sit for state-mandated assessments, each school 
would choose one or more assessments—whether 
traditional, objective assessments; portfolios of work; 
measures of social or emotional gains; or rubric- 
based grading of more subjective, performance-based 


assessments—that are valid, reliable, and appro- 
priate for the program of study. Schools with com- 
mon missions and purposes would either choose 
the same set of assessments, to allow parents and 
policymakers to make valid comparisons, or create 
ways to compare locally offered assessments by 
giving subsets of their student population a sample 
of questions from a common assessment. 

All programs would be required to administer 
pre- and post-assessments to students so they can 
report individual student growth, and those assess- 
ments must be taken under identical conditions. 
The assessments would test items below and above 
grade level so they accurately capture each individ- 
ual student’s starting point and growth. Pre- 
assessments upon enrollment are crucial and 
should be required, as district schools where stu- 
dents were formerly enrolled often don’t transfer 
students’ academic records to their new school. 

Ideally, the schools would use not just norm- 
referenced tests," which measure growth relative 
to other students, but also criterion-referenced 
growth assessments, which measure performance 
relative to a specific set of standards (like those 
that i-Ready offers’) to report the results in ways 
that are transparent, the public can understand, 
and capture raw results around student progres- 
sion. Statisticians can then use the raw results to 
make cohort-based comparisons to inform various 
rankings and comparisons between schools or to 
make policy decisions. In addition, alternative 
schools focused on serving students who have 
dealt with trauma should administer up-front risk 
assessments and survey measures around social- 
emotional learning, school climate, and student re- 
lationships. 

Finally, instead of assuming that average daily 
attendance and other seat-time-based measures 
correlate to learning, to the extent schools must 
continue these measurements to receive funds, 
they should seek waivers so they can shift to en- 
gagement measures that track student progres- 
sion. These could include the use of and comple- 
tion or submission of assignments. 


Program Completion. Although completion is a 
measure that can be easily manipulated—as it’s 
relatively easy to print a diploma for a student 
compared to ensuring they’ve learned, and it falls 


AMERICAN ENTERPRISE INSTITUTE 


under the category of an output more than an 
outcome—it’s useful for parents trying to under- 
stand the likelihood of their child successfully 
graduating from a given school. One adjustment to 
make to measure completion is to survey students 
and parents upon enrollment to understand their 
intention in enrolling. 

Because many families enroll their children in 
alternative settings out of a desire to help them 
catch up and then return to traditional school; 
temporarily escape a bullying situation, a bad 
teacher, or a bad year; provide a stopgap during a 
family move between districts or states; or avoid 
poor remote learning (as we saw this past year), 
it’s helpful to allow those families to understand 
whether the school is working for learners who 
have similar intentions. Do the incoming students 
intend to stay for a short time to transition to 
another school? Do they intend to stay for only the 
year? Do they intend to stay through graduation? 
In turn, these students could then be tracked by 
whether they successfully complete the term they 
intended upon enrollment. These reasons for 
enrolling can change, of course, but it’s helpful to 
create cohorts based on entering intentions so that 
parents can make decisions based on families with 
similar profiles. 


Placement upon Leaving. Schools should report stu- 
dents’ placement information upon exiting their 
program. If the students left early, did they transfer 
to another school? Did they drop out? If the stu- 
dents graduated, where did they go next? For high 
schools, understanding where students went after 
leaving—for example, to a job, a two-year or four- 
year program, or a shorter-term training pro- 
gram—is helpful information for accountability to 
work. This would assure prospective parents and 
students and policymakers that attending the 
school helped students achieve their goals, what- 
ever those might be. 


Earnings After Graduation. For most schools, earn- 
ings after graduation will not be a relevant metric, 
but for schools with a career and technical educa- 
tion perspective, they might be. This measure would 
capture students’ salaries upon leaving the pro- 
gram and entering the workforce. In addition, for 
students graduating from career-focused schools, 


policymakers could tie earnings data over time to 
those schools to build longitudinal views of how 
the schools prepared students for not just their 
first job but their career. 


Satisfaction and Confirmation of Purpose. All pro- 
grams would be required to understand whether 
the school’s mission is consistent with why parents 
and students chose the school and then to measure 
whether parents and students were satisfied with 
the experience retrospectively. For the former 
objective, the key will be to understand the par- 
ents’ and students’ goals when they enroll. 

This information is vital because many virtual 
schools, for example, have argued that they have 
low completion rates or low test-taking rates because 
their student population has a different set of goals 
when they enroll, which their internal data sup- 
port. When that’s the case, it’s important to cap- 
ture upon intake, not at the end of a student’s 
experience in a program. 


Many virtual schools, for example, 
have argued that they have low 
completion rates or low test-taking 
rates because their student popula- 
tion has a different set of goals 
when they enroll. 


The above five categories can then be used 
alongside demographic classification and entering 
academic level (e.g., students who are more than 
48 months behind academically) to disaggregate 
outcomes and measure success for each group 
based on whether they attained their entering goal. 
Students should also be surveyed after they leave 
to understand their satisfaction. This measure 
should be based on former students answering the 
following question on a 10-point scale: “Knowing 
what you know now, would you choose to repeat 
your experience here?”4 

Although many programs are fond of asking 
such questions with a net promoter score while 
students are enrolled, this measurement has two 
key challenges. First, providers can effectively 
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manipulate students’ answers by providing perks 
around the time the survey is documented—be 
that pizza parties or good grades. Second, because 
the quality of an education can be relatively opaque 
to learners, it can be hard for students to know 
whether their education will pay off in the medium 
or long term or how it compares to other options 
in which they might have enrolled. 

Consequently, this question is better asked 
after students have left the program for some 
length of time and can reflect with the benefit of 
hindsight and more context on the quality of their 
experience. One factor with which policymakers 
and researchers will need to grapple is how to think 
about response rate in these surveys and, in partic- 
ular, how to judge a low response rate. For exam- 
ple, if only 10 percent of alumni respond to a sur- 
vey and report high satisfaction, is that because 
only those who were enthusiastic about the expe- 
rience decided to reply? Or is it truly representa- 
tive of the broader student experience? 

As an interim measure, schools could choose 
to also administer school climate surveys, which 
providers such as Panorama Education offer,'5 to 
gain a more current measure of the schooling 
experience. 


Data Should Be Audited. All the measures on 
which a school collects data and reports outcomes 
consistent with a set of standards should be 
audited. Auditors should make sure providers 
don’t manipulate the test conditions under which 
students take exams, for example, and ensure that 
work a school represents as a given student’s was 
in fact done by that student. Auditors should not 
only look at the data a school produces but also 
do independent verifications with students and 
parents annually to provide assurance that the 
outcomes measured are accurate. 

With such an audit process in place, it’s important 
to avoid conflicts of interest between schools and 
the auditor, which could range from traditional 
accounting firms that conduct audits in other areas, 
such as compliance with environmental standards, 
to nonprofit entities trained in the Education 
Quality Outcomes Standards. To avoid such con- 
flicts, education can look to the rules governing 
financial auditing of publicly traded companies 
as a way forward. 


How It Might Look 


Against this framework, a full-time virtual school might measure the following set of indicators, which 
regulators and others compiling rankings could choose to weight differently. 


Learning Outcomes 


e Have 80-plus percent of learners take state-mandated assessments, 


e Measure individual student learning using i-Ready assessments, 


e Measure student social-emotional learning growth through up-front risk assessment and periodic 


surveys, and 


e Use individual students’ project portfolios to assess mastery and progress against competencies 
through the use of Mastery Transcript Consortium,"* a nonprofit that offers an alternative framework 


to document student learning in high school. 


Completion 


e Measure the annual percentage that successfully completes the desired outcome based on student 


intention. 
Satisfaction 


e Administer school climate surveys, 


e Gather retrospective student satisfaction responses, and 


e Ask for parent satisfaction responses (same format as the student satisfaction question). 


Specifically, the Sarbanes-Oxley regulatory 
regime, which Congress passed and President 
George W. Bush signed in 2002 to help protect 
investors from fraudulent financial reporting, has 
set the gold standard for auditing claims of 
performance.’”? Although evidence suggests the 
Sarbanes-Oxley Act has stifled small businesses 
and innovation because of its reporting require- 
ments (and associated costs),!* the reasons for 
the protections it has established are, in certain 
key respects, similar to the reasons for protec- 
tions involving auditing educational institutions’ 
performance claims. As such, its rigorous set of 
standards for monitoring and reporting on an 
entity’s claims around outcomes seems appropri- 
ate for the education sector. 

Three key principles from the Sarbanes-Oxley 
law ought to apply. First, the entity that evaluates 
an educational program’s claims about outcomes 
must be independent of the program, which means 
that the entities must not have a deeper economic 
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affiliation. There must not be a vendor or con- 
sulting relationship between the auditor and the 
school, and the auditor must not have a direct 
economic stake in the school. Membership groups, 
in which schools pay to join (akin to accreditors), 
are likewise not in a position to conduct audits. 
And there must be no family ties between an audi- 
tor and a school. 

Second, the program being evaluated must not 
be in a position to exercise undue influence on the 
evaluating entity. This means there must be no 
past employer relationship between the auditor 
and the school, and schools should rotate their 
auditors every five years. 

Finally, the entity that evaluates an educational 
program’s claims about outcomes must take eco- 
nomic risk in excess of the cost of the expected 
audit revenue. This means the auditor must be on 
the hook for material losses if it engages in wrong- 
doing with the audit of a school. 


Conclusion attend them as opposed to attending traditional 
district schools), such a system would allow par- 
ents to identify which schools are best serving 
specific student needs so they can find the right 
match for their situation. That transparency will 
create more accountability in the system and allow 
regulators to compare schools serving similar 
populations of students with similar missions 
against a robust set of metrics. And it would allow 
researchers to better correlate short-term measures 
with longer-term outcomes to improve our causal 
understanding of what leads to success for whom. 


Ultimately, an outcomes-based model of account- 
ability for alternative schools should do a few 
things. It should encourage schools to innovate to 
better serve their students. And it should give them 
enough breathability in the metrics on which they 
report that what they measure makes sense with 
their mission but doesn’t allow them to duck from 
the hard task of serving students well and appro- 
priately. 

Because alternative schools are typically schools 
of choice (i.e., parents and students choose to 
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